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Process  instrumentation  provides  trainers  and  testers  with  deeper  insight  into  the  activities, 
both  human  and  technology,  that  affect  system  and  warfighter  performance  and  mission 
effectiveness.  To  this  end,  the  authors  have  designed  and  developed  several  process 
instrumentation  systems,  working  with  both  U.S.  Air  Force  warfighter  trainers  and  testers. 

Both  trainers  and  testers  need  to,  in  operationally  relevant  environments,  reconstruct  events  to 
meet  teaching  needs  or  testing  requirements.  These  similar  reconstruction  activities  drive  the 
need  for  a  method  to  capture  human,  as  well  as  technology,  activities.  This  article  describes 
process  instrumentation  systems  and  their  benefits  for  trainers  and  testers  and  then  describes, 
with  examples  and  a  case  study,  two  process  instrumentation  systems. 
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In  training  and  test  environments,  the  instru¬ 
mentation  of  user-operator  work  processes 
and  communications,  in  addition  to  the 
instrumentation  of  technology  (e.g.,  comput¬ 
er  systems  and  networks),  can  provide  new 
and  needed  insights  into  mission  effectiveness.  In 
training  environments,  trainers  can  use  the  information 
collected  and  displayed  by  process  instrumentation  to 
reconstruct  training  events  and  augment  after-action 
reviews,  highlighting  key  trainee  decisions.  Similar 
information  and  reconstructions  can  assist  operational 
testers  in  determining  a  system-under-test’s  impact  on 
total  performance  and  mission  effectiveness. 

During  both  training  and  test  events,  trainers  and 
testers  develop  scenarios  using  Master  Scenario  Event 
List  (MSEL)  inputs.  Both  use  these  MSELs  to 
stimulate  desired  outcomes  for  focused  observation, 
analysis,  and  debrief  and  reporting.  Training  after¬ 


action  reviews  rely  on  reconstruction  technology  and 
human  activities  to  determine  student  performance  and 
reinforce  teaching  points.  Test  analysis  of  MSEL  test 
inputs  involves  the  reconstruction  of  technology  and 
human  activities  to  determine  the  efficacy  of  the 
system  or  network  under  test. 

Over  the  past  5  years,  the  Johns  Hopkins  University 
Applied  Physics  Laboratory  (JHU/APL)  has  devel¬ 
oped  several  work  process  and  personnel  communica¬ 
tion  instrumentation  systems.  These  systems  have  been 
used  by  the  Time  Sensitive  Targeting/Dynamic 
Targeting  trainers  at  the  U.S.  Air  Force’s  Combined 
Air  and  Space  Operation  Center — Nellis  (CAOC-N) 
and  researchers  at  the  Air  Force  Research  Laborato¬ 
ry — 711th  Human  Performance  Wing  for  several 
training  research  exercises  focused  on  dynamic  effects, 
as  well  as  assessors  of  the  Warplan-to-Warfighter 
Forwarder  (WWF)  Spiral  II  initiative  during  the  Air 
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Force’s  Joint  Expeditionary  Force  Exercise  (JEFX)  09- 
3.  In  addition,  the  U.S.  Joint  Forces  Command  has 
incorporated  several  JHU/APL  instrumentation  capa¬ 
bilities  into  their  Joint  After- Action  Review  Resource 
Library. 

This  article  provides  an  overview  of  the  benefits  of 
work  process  and  personnel  communication  instru¬ 
mentation  in  both  training  and  test  environments,  a 
description  of  the  instrumentation  systems  developed 
by  JHU/APL  with  examples  of  their  use  by  CAOC-N 
trainers,  and  a  case  study  of  WWF  assessors  during 
JEFX  09. 

Benefits  of  work  process  instrumentation 

Process  instrumentation  systems  automatically  col¬ 
lect,  organize,  and  archive  large  amounts  of  work 
process  and  personnel  communications  data,  as  well  as 
pertinent  technology  data.  During  events,  instead  of 
trainers  and  testers  manually  searching  and  recording 
the  information  they  will  later  use  for  reconstructions 
and  analyses,  process  instrumentation  systems  auto¬ 
matically,  and  with  minimal  intrusiveness,  collect  work 
process,  communications,  and  system  data.  Collected 
data  are  organized  and  archived  in  databases,  making  it 
easily  accessible  to  performance  analysis  and  assess¬ 
ment  systems,  as  well  as  searchable  with  user-specified 
criteria.  Such  instrumentation  significantly  improves 
the  ability  of  trainers  and  testers  to  evaluate  overall 
mission  effectiveness,  assess  human-centric  issues,  such 
as  effectiveness  of  Command  and  Control  (C2),  and 
contribute  to  Tactics,  Techniques,  and  Procedures 
(TTP)  development. 

In  addition  to  relieving  trainers  and  testers  of  the 
burden  of  manually  collecting  data  in  real  time  during  an 
event  and  the  danger  of  missing  key  information  because 
of  trainer  or  tester  task  overload  or  data  inundation,  the 
persistent  archive  of  data  by  process  instrumentation 
systems  creates  a  historical  record  of  events  and 
activities.  Nowadays,  all  too  often,  the  work  process, 
communications,  and  system  data  during  training  and 
test  events  is  lost  after  the  event  ends  or  is  captured  in 
nondigital,  not  easily  accessible  Information  Technol- 
ogy  (IT)  system  formats,  like  Microsoft  Office  docu¬ 
ments  or  proprietary,  stove-piped  data  sources.  Without 
historical  data  to  establish  process  and  mission  effec¬ 
tiveness  baselines,  the  benefits  of  training  programs  and 
new  systems  under  test  become  difficult  to  determine 
and  are  often  subjectively  assessed. 

The  ability  to  archive  data  and  refer  to  it  later  also 
enables  trend  analysis.  This  can  be  particularly  useful  in 
the  training  environment  to  gauge  students’  progress  in 
the  training  process.  Trend  analysis  can  also  provide 
insights  into  effectiveness  of  training  procedures  as 
well  as  improvements  over  time  of  a  technology  being 


tested — given  analysis  is  conducted  to  isolate  contri¬ 
butions  to  effectiveness  of  specific  material  systems, 
processes,  or  learning. 

Process  instrumentation  systems  also  provide  the 
capability  to  reconstruct  both  technology  and  human 
activities,  using  empirically  based  “truth”  data,  that  is, 
data  from  IT  systems  used  by  humans  to  complete  an 
activity  or  process,  as  opposed  to  notes  or  memories  of 
the  participants,  trainers,  and  testers.  A  reconstruction 
can  take  many  forms  but  is  most  useful  if  it  contains 
both  technology  actions  and  human  actions  that  have 
been  correlated  and  can  be  examined  through  specific 
events  (e.g.,  MSEL  items)  in  a  larger  mission  context, 
often  called  a  thread.  With  the  activities  correlated  and 
focused  on  a  single  event  thread,  testers  and  trainers 
can  better  understand  the  timing  and  process  actions  to 
evaluate  the  performance  impact  of  the  MSEL  input 
on  the  overall  mission.  Additional  context  from  other 
threads  provide  insights  into  how  C2  (e.g.,  leadership, 
team  coordination,  and  decision  making)  and  TTP 
positively  (or  negatively)  influenced  performance 
outcomes.  With  the  technology  and  human  activities 
correlated,  attention  is  focused  on  total  performance 
and  mission  effectiveness,  as  opposed  to  just  warfight¬ 
ers’  opinions. 

Based  on  extensive  JHU/APL  observations,  event 
reconstruction  is  often  performed  manually,  consuming 
much  time  and  labor,  and  limiting  the  amount  and 
depth  of  analysis.  Automating  the  collection  and 
analysis  of  user-operator  process  data  and  communi¬ 
cations  results  in  a  reduction  in  time  and  labor  as  well 
as  improved  quality  in  reconstruction  and  analysis. 
Existing  instrumentation  systems,  such  as  the  Air 
Combat  Maneuvering  Instrumentation  and  the  Nellis 
Air  Combat  Training  System  used  for  tactical  aircrew 
training,  support  this  position. 

A  distinction  needs  to  be  made  between  IT  system 
instrumentation,  which  focuses  on  data  and  system 
operation  validation,  and  process  instrumentation. 
Process  instrumentation  brings  the  human  aspects  of 
a  system,  such  as  workflows,  decision  making,  and 
collaboration  and  communication,  into  focus  for  the 
training  and  testing  communities  and  allows  a  full 
accounting  of  total  system  performance  and  mission 
effectiveness. 

Process  instrumentation  systems:  CPAS 
1.2  and  1.3 

To  research  and  exploit  the  benefits  of  process 
instrumentation,  JHU/APL  developed  two  systems, 
the  CAOC  Performance  Assessment  System  (CPAS) 
vl.2  and  CPAS  vl.3  (or  eCPAS  for  Enhanced  CPAS). 

JHU/APL  built  CPAS  vl.2  for  the  CAOC-N 
trainers.  This  version  of  CPAS  collects  work  process 
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Figure  1.  CPAS  composite  and  drill-down  views. 


data  from  the  Joint  Automated  Deep  Operations 
Coordination  System  QADOCS)  mission  collabora¬ 
tion  technology  and  communications  data  from  several 
text  chat  collaboration  systems.  CPAS  stores  the 
collected  data  in  a  relational  database  and  displays  it 
visually  using  timelines.  Other  capabilities  include 
filtering  and  searching  the  collected  communication 
messages,  correlating  key  process  events  with  commu¬ 
nication  messages,  and  detecting  process  anomalies 
using  user-configured  parameters. 

A  later  version  of  CPAS,  Enhanced  CPAS  (eC- 
PAS),  was  customized  to  meet  testing  requirements 
from  the  Air  Force’s  605th  Test  and  Evaluation 
Squadron.  eCPAS  added  the  capabilities  to  collect 
Link- 16  messages,  computer  screen  displays,  and  voice 
communications,  as  well  as  transcribe  voice  communi¬ 
cations  into  text  for  improved  search  and  correlation. 
Multiple  data  sources,  including  voice,  Link-16,  chat, 
and  JADOCS  remarks,  may  be  searched  simultaneous¬ 
ly  and  displayed  in  user-customizable  timelines. 

CPAS  version  1.2 

CPAS  was  delivered  to  the  Air  Force  in  2006.  The 
primary  purpose  of  the  CPAS  instrumentation  system 
was  to  support  individual  and  team  training  of  the  Air 
Force’s  Air  Operations  Center’s  dynamic  targeting 
(DT)  processes.  Operational  trainers  use  CPAS  to 
monitor  in  near-real  time  individual  and  collective 
trainee  performance.  They  compare  observed  perfor¬ 
mance  against  expectations,  especially  in  relation  to  the 
lesson  objectives,  MSEL  events,  and  other  stimuli.  By 


monitoring  these  activities  in  near-real  time,  trainers 
can  flag  critical  learning  opportunities  for  later  analysis 
and  debriefing,  collaborate  with  others,  and  frame  their 
initial  understanding  of  individual  and  collective  trainee 
performance.  Using  CPAS  performance  feedback,  the 
trainers  and  event  managers  can  also  monitor  and 
manipulate  the  event  timeline  to  assess  progress  toward 
achieving  lesson  or  exercise  objectives.  This  includes 
monitoring  the  CPAS -generated  JADOCS  timeline 
(e.g.,  workflow  manager  playbacks)  and  warfighter  chat 
systems  for  correlated  activity.  Finally,  trainers  use 
CPAS  to  support  training  debriefs  after  the  training 
events  and  reconstruct  student  events  and  activities. 

CPAS  uses  a  combination  of  composite  activity 
views  (see  top  of  Figure  1 )  and  drill-down  focus  views 
(see  bottom  of  Figure  1).  The  timescale  is  located 
along  the  top  of  the  view,  with  each  major  event  thread 
indicated  by  the  track  number,  event  name,  and 
priority.  In  Figure  1,  the  trainer  is  exploring  the  DT 
Chief  s  SCUD  convoy  chat  activity  alongside  the  SA-6 
TEL  targeting  process.  Note  the  amount  of  detail 
CPAS  displays  (see  Figure  2),  mapping  actual  events 
onto  the  U.S.  military’s  joint  kill  chain  process  steps: 
Find-Fix-Track-Target- Engage-Assess  (F2T2EA). 

Through  interactions  with  Air  Force  personnel, 
JHU/APL  realized  that  different  training  groups  and 
warfighting  regions  apply  region-  or  mission-specific 
approaches  to  their  process  steps.  As  a  result,  CPAS 
allows  users  to  tailor  each  of  the  major  process  views. 
Figure  2  shows  the  basic  F2T2EA  process  with 
discrete  color-coded  steps,  starting  and  ending  times 
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Figure  2.  CPAS  F2T2EA  depiction  of  single  mission. 


defined  as  part  of  the  CPAS  configuration  file  for  this 
unique  training  event. 

CPAS  was  designed  for  realistic,  dynamic  scenarios 
and  active  instructor  involvement  in  the  training 
process.  The  system  allows  instructors  to  add  their 
own  notes,  based  on  observations,  trainee  performance 
issues,  or  other  potential  learning  opportunities,  to  the 
CPAS  database. 

Based  on  CAOC-N  trainer  requirements,  CPAS 
exports  relevant  data  fields  into  a  Microsoft  Excel 
spreadsheet  for  further  analysis  and  development  of 
debrief  handouts. 


The  major  analysis  function  in  CPAS  is  a  work- 
process  anomaly  detector  (see  Figure  3).  Based  on 
“school  solution”  process  activities  and  timelines, 
trainers  can  compare  individual  and  team  trainee 
performance  against  the  norm.  In  Figure  3,  an  intuitive 
display  enables  the  trainer  to  recognize  deviations  from 
the  expected  work  processes,  providing  instruction 
points  for  trainers  to  use  during  feedback  sessions  to 
the  training  audience. 

CPAS  augments  trainer  analysis  and  trainee  perfor¬ 
mance  feedback  with  a  comprehensive  JADOCS 
workflow  manager  playback  feature  (see  bottom  of 


Figure  3.  CPAS  process  anomaly  display. 


31(2)  •  June  2010  267 


Ockerman  et  al. 


Figure  4.  CPAS  JADOCS  manager  playback  display. 


Figure  4).  Using  the  slider  bar  at  the  bottom  of  the 
figure,  the  user  can  “playback”  and  observe  activity  and 
state  changes  in  the  respective  JADOCS  managers 
over  time. 

Based  on  JHU/APL’s  analysis  of  warfighter  DT 
processes  and  TTP  documentation,  CPAS  1.2  focused 
on  the  chat  environment  as  the  primary  context 
variable.  This  gave  the  instructor  an  increased 
understanding  of  “what’s  going  on”  among  DT  team 
members  during  a  mission.  Figure  5  shows  the  wide 
variety  of  search  and  filter  functions  available  for  the 
trainer  to  analyze  chat  communications. 


By  collecting  JADOCS  and  chat  transactions, 
CPAS  provided  trainers  with  the  ability  to  easily 
correlate  these  two  disparate  data  sources  for  their 
trainee  performance  analysis  and  debriefing.  Figure  6 
illustrates  CPAS’s  correlation  capability. 

The  capabilities  CPAS  provides  to  Air  and  Space 
Operations  Center  (AOC)  DT  trainers  is  enabling  a 
transformation  in  the  way  AOC  DT  training  is 
accomplished.  CPAS  reduces  the  subjective  assess¬ 
ments  of  trainers  and  students,  and  creates  more 
reliable,  objectively  recorded  and  measured  facts. 
Already,  CPAS  functions  as  a  training-force  multiplier. 
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Figure  6.  CPAS  timeline-chat  correlation  displays. 


No  longer  are  trainers  required  to  spend  valuable 
training  time  manually  collecting  data  for  reconstruction. 
Instead,  trainers  have  more  time  to  observe,  interact,  and 
assess.  Although  not  formally  evaluated,  the  trainers 
have  estimated  that  the  use  of  CPAS  has  provided  a  75% 
reduction  in  the  amount  of  time  required  to  prepare  a 
debrief.  Furthermore,  CPAS  has  enabled  the  training 
debrief  process  to  evolve  from  filling  in  key  event  times 
in  a  Microsoft  Excel  spreadsheet  to  spending  more  time 
analyzing  the  training  threads  and  understanding  what 
happened,  when,  and  why. 

CPAS  version  1.3 

CPAS  1.3,  or  eCPAS,  was  delivered  to  the  Air  Force 
in  2009.  The  primary  purpose  of  the  eCPAS  instru¬ 
mentation  system  was  to  support  operational  testing,  as 
well  as  continue  support  for  individual  and  team  training 
of  several  of  the  AOC’s  combat  operations  division 
processes.  eCPAS  enables  operational  testers  to  monitor 
mission  thread  activities  under  test  in  near-real  time. 
They  can  compare  observed  activities  and  events  against 
expectations  outlined  in  the  test  plan.  By  monitoring 
these  activities  in  near  real-time,  they  can  flag  critical 
observations  for  later  analysis,  collaborate  with  others, 
and  frame  their  initial  understanding  of  tested  system 
performance  in  the  context  of  the  mission  conditions, 
processes,  and  systems  used. 

Building  on  the  successes  of  CPAS,  eCPAS  added 
several  valuable  capabilities.  eCPAS  retained  the  JA- 
DOCS  timelines  and  workflow  manager  playbacks,  as 
well  as  the  warfighter  chat  systems  for  correlated  activity. 


Based  on  operational  tester  requirements,  eCPAS  added 
the  collection  of  additional  chat  systems,  voice  on  C2 
audio  channels,  and  selected  Link-16  messages.  Testers 
can  also  identify  significant  observations  via  eCPAS  by 
inserting  flags  directly  onto  the  user  interface.  Flags  can 
include  both  comments  and  screen  captures  to  provide 
primary  source  data  for  anomaly  analysis.  The  test  team 
can  take  an  instantaneous  screen  capture  or  set  up 
periodic  capture  for  the  duration  of  the  test  event. 

eCPAS  has  a  “unified  search”  to  support  its 
correlation  and  analysis  capability.  Figure  7  shows  the 
increased  fidelity  of  data  capture  that  distinguishes 
eCPAS  from  CPAS.  Testers  can  selectively  search  chat 
systems,  rooms,  and  users;  data  link  sources;  and  voice 
data  sources.  Testers  can  also  query  JADOCS  data  to 
view  specific  transactions  from  JADOCS’  mission 
history  files.  In  addition,  testers  can  filter  by  keywords 
and  times  across  all  the  selected  communications 
simultaneously.  Once  the  user-defined  search  data  are 
returned,  three  different  keywords  can  be  highlighted 
throughout  the  data  to  show  relevant  trends  and  pick 
out  threads.  Finally,  user-defined  search  data  can  be 
sent  to  a  new  timeline  for  display  and  analysis. 

eCPAS  improves  on  the  CPAS  framework  for  the 
JADOCS  mission  manager  playbacks  (see  Figure  8)  by 
making  it  easier  for  testers  to  configure  and  update  the 
manager  displays  in  real  time. 

Testers  can  inject  events  into  the  eCPAS  database  by 
generating  process  “flags”  at  any  time  during  test  event 
planning  or  execution  (see  Figure  9).  Flags  allow  the 
testers  to  integrate  their  notes  and  observations  with  the 
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Results 

Previous  Searches: 


DateTime 

MessageType 

Source 

Message  A 

03/10/09 14:40:35 

Voice:  WCAS2 

Viper3 

yeah  go  ahead  (0.7895) 

03/10/09  14:40:42 

Chat:  IRC  Log 

SI  DO:  COD_ALL 

CCO  :  Would  you  like  to  start  tracking  with  PRed? 

03/10/09  14:40:48 

Link:  12.0 

10- 

//MsgNum  3/l_Wrd_Fmt  0/J_Label  12/J_Sublabel  0/MLI  0/' 

03/10/09 14:40:49 

Link:  12.6 

10- 

//MsgNum  12/l_Wrd_Fmt  0/J_Label  12/J_Subiabel  6/MU  7/ 

03/10/09 14:40:57 

Chat:  IRC  Log 

CCO:  COD_ALL 

SIDO  :  No  we  donl  have  resources 

03/10/09  14:41:05 

Voice:  WCAS2 

Magic 

four  (0.7755) 

03/10/09  14:41:05 

Voice:  WCAS2 

Viper2 

two  (0.3015) 

03/10/0914:41:06 

Voice:  WCAS2 

Magic 

viper  one  single  group  buliseye  zero  five  zero  fourteen  twenty  fourthous 

03/10/0914:41:12 

Chat:  IRCLog 

CCO:  COD_ALL 

Listen  up  everybody 

03/10/09  14:41:13 

Voice:  WCAS2 

Viperl 

viper  one  (0.8195) 

03/10/09  14:41:18 

Voice:  WCAS2 

Vrper3 

viper  three  still  spin  cold  (1 .3075) 

03/10/09 14:4122 

Voice:  WCAS2 

Vrper3 

three  and  four  come  left  (0.8835) 

03/10/0914:41:27 

Chat:  IRCLog 

CCO:  COD_ALL 

we  have  a  launch  at  3234.2343  by  2343.2343 2 

03/10/09  14:41:30 

Voice:  WCAS2 

Viperl 

viper  one  and  two  reference  two  six  zero  (2.7395) 

03/10/0914:41:41 

Voice:  WCAS2 

Viperl 

viper  one  gate  (1 .2795) 

03/10/0914:41  42 

Chat:  IRCLog 

CCO:  COD_ALL 

I'm  definitely  feeling  a  bit  drugged  and  woozy 

03/10/0914:41:57 

Chat:  IRCLog 

TSTChief :  COD_ALL 

CCO  :  Maybe  you  should  get  some  rest 

03/10/09  14:42:12 

Chat:  IRCLog 

SIDO:  COD_ALL 

CCO:  Yeah  some  R&R  would  be  good,  no? 

03/10/09  14:42:27 

Chat:  IRCLog 

CCO:  COD_ALL 

I'm  not  really  sure  what  1  need 

03/10/09 14:42:30 

Remaik:  TCT 

CCO:  BA0041 

131442ZSEP2005-30  -  NCAOC  -  CCO~'This  is  a  new  tasking'-" 

03/10/09 14:42:30 

What:  TCT:  ACC  Package  Id 

BA0041 :  CCO 

->GF 

03/10/09  14:42:30 

What:  TCT:  ACC  Call  Sign 

BA0041:CCO 

->  VIPER  31 

03/10/09  14:42:30 

What:  TCT:  ACC  Mission  No 

BA0041  CCO 

->33456  v 

Records  Founds:  1412 

|  l*Pg 


I  |lrack 


]  IWl  I 


Show  on  Timeline 


03:54:08  Mar  01,  2010  Client  Mode:  Server  Connected  Database:  \\kotemtl  wv2\ECPAS  Event  Test  Event  1  Session:  First  Session  Location:  APL  User  Profile:  Demo 


Figure  7.  eCPAS  unified  search  control  display. 


Figure  8.  eCPAS  JADOCS  manager  playback  display. 
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H  4  |  2  of  2  |  X  d  F«lter ;  convoy  Property  »  Apply 


Name  StartTime  EndTime  Comments 

Convoy  Started  as  Expected  14:02:42  03/10/09  14:02:42  03/10/09  Convoy  left  base  as  planned. 


Figure  9.  eCPAS  “flag”  entry. 


rest  of  the  data  being  collected  by  eCPAS.  Multiple 
testers  can  create  and  edit  these  flags,  which  are  viewable 
by  the  entire  test  team.  Flags  can  be  filtered  (see  bottom 
of  Figure  9)  for  quick  access  and  editing. 

Another  capability  added  in  eCPAS  allows  testers  to 
view  timelines  of  data  alongside  one  another.  This  new 
view,  the  Command  and  Control  Process  Common 
Operating  Picture,  or  “C2  Process  COP,”  is  shown  in 
Figure  10.  The  aggregate  activity  in  JADOCS,  chat, 
Link-16,  and  observer  flags  in  area  (1)  shows  the 
correlation  of  events  in  these  different  domains  over 
time.  This  view  would  logically  trigger  the  tester  to 
conduct  further  analysis  in  this  time  slice  and  also 
provide  the  “at  a  glance”  overview  for  the  test  team 
leadership.  Conversely,  the  proliferation  of  observation 
flags  (2)  over  a  greater  time  slice  might  cause  the  tester 
to  question  why  no  activities  have  been  recorded  in  the 
other  (e.g.,  JADOCS,  chat,  Link-16)  domains. 

Table  1  provides  an  overview  of  the  CPAS  and 
eCPAS  data  sources,  and  the  information  they  provide. 
In  addition,  it  provides  a  one-sentence  summary  of 
each  version’s  targeted  use. 


JEFX-09  case  study 

The  goal  of  the  Warplan-to-Warfighter  Forwarder 
Spiral  II  (WWF  II)  system  was  to  enable  JADOCS, 
Target  Package  Generator,  and  Network  Enabled 
Weapons  Control  Interface  Manager  to  send  ma- 
chine-to-machine  targeting  data  from  the  AOC 
directly  to  the  airborne  C2  aircraft,  combat  aircraft, 
or  Network-Enabled  Weapons  (NEW)  using  Link-16 
standards  (MIL-STD-6016D)  and  allow  automated 
status  updates  in  a  dynamic  targeting  environment. 
The  task  being  performed  by  the  team  assessing  WWF 
II  in  JEFX  09-3  was  to  analyze  the  timeliness, 
accuracy,  and  completeness  of  all  the  J-series  messages 
that  were  transmitted  on  the  network.  Although  not  a 
formal  operational  test  event,  the  JEFX  09-3  experi¬ 
ment  was  a  suitable  surrogate  that  included  real 
operators  and  aircrew  conducting  operations  in  a  live, 
virtual,  and  constructive  environment.  Live  aircraft  (F- 
15Es,  F-16CMs,  a  B-2,  and  an  E-3  AWACS)  flew 
missions  in  the  Nellis  ranges;  simulated  aircraft,  both 
virtual  (E-8  JSTARS)  and  constructive  (F-15E,  F- 
16C),  were  located  at  Eglin  Air  Force  Base  (AFB),  FL; 


31(2)  •  June  2010  271 


Ockerman  et  al. 


iretagSTI  ^  -  No  Wpn/Tgt  Ptuxng  Q  -  Wpn/Tgi  f’oiong  Q.  ta’.k.yl  L»0  Floor  B-  MSN  Rotoll  j~  -  SuippoO 
UX4kl7X4v  rv  NMO  MxtXXr'HwllhBrlllxiUrtl  M*ixt  HiWxIllfW'  rtWHWtfcwHn  iLi^fcx.HW'  Umorndfc  tm«;  InJ  IvnM  I  Vi 


Figure  10.  eCPAS  C2  Process  COP  for  visual  correlation  and  event  overview. 


and  a  live  AOC  manned  by  real  operators  was  in  place 
at  Nellis. 

The  primary  technology  used  by  one  member  of  the 
assessment  team  to  monitor,  in  near-real  time,  the 
WWF  II  system  during  JEFX  09-3  was  JADOCS. 
The  JADOCS  workflow  managers  (Joint  Time 
Sensitive  Target  Manager  and  the  Intra-AOC  Man¬ 
ager)  together  with  the  Coordination  and  Target  Data 
tabs  in  each  mission  folder  provided  most  of  the  data 
needed  to  maintain  an  understanding  of  the  mission 
information  flow.  Other  technologies  used  to  build  and 
maintain  situational  awareness  during  live-fly  opera¬ 
tions  included  the  CPAS  instrumentation  displays,  the 
XMPP  chat  client  (transverse),  the  multilink  transla¬ 
tional  and  display  system,  the  tactical  view  command 
and  control,  the  joint  windows  warfare  assessment 
model,  and  audible  voice  communications. 

During  both  live-fly  and  modeling  and  simulation 
operations  in  the  experiment,  the  event  summary 
timeline  display  of  CPAS  version  1.2.5. 3  was  used  to 
maintain  insight  into  the  state  of  each  of  the  15  to  20 
missions  that  had  been  tasked.  Multiple  missions  were 
prosecuted  simultaneously;  therefore,  the  event  sum¬ 
mary  timeline  feature  helped  the  assessor  to  keep  track 
of  the  missions  and  key  events  as  they  occurred.  When 
required,  further  details  of  a  specific  mission  were 
available  by  drilling  down  to  the  detailed  mission  phase’s 
display  from  the  event  summary  window.  These  drill¬ 
down  views  allowed  analysis  of  key  events  in  the  context 
of  the  phases  defined  by  WWF  II  assessors. 


Another  useful  CPAS  feature  was  the  JADOCS 
workflow  manager’s  playback  displays.  Using  the  Intra- 
AOC  Manager  playback,  an  assessor  could  quickly  find 
when  the  critical  columns  changed  status  by  viewing 
changes  in  the  color  and/or  three-letter  status  code  of 
the  manager’s  display  lights,  known  as  “chicklets.”  Many 
times  during  the  exercise,  the  JADOCS  Intra-AOC 
Manager  chicklets  (which  are  incredibly  small  on  an 
extremely  crowded  computer  monitor)  changed  very 
quickly  and  an  assessor  had  to  be  observing  the  specific 
chicklet  at  the  exact  moment  to  verify  the  transitory 
status.  With  the  playback  manager  in  CPAS,  however, 
the  assessor  could  easily  go  back  and  step  through  the 
scenario  to  verify  which  actual  transactions  occurred  at 
which  time.  The  CPAS  chat  search  also  proved  useful 
by  allowing  different  chat  rooms  and  roles  to  be  selected 
and  then  searched  around  a  specific  timeframe  or  for 
specific  keywords.  Thus,  CPAS  helped  in  tracking  the 
various  human  and  machine  actions  and  chat  commu¬ 
nications,  helping  assessors  reconstruct  and  understand 
the  exact  interactions  among  the  systems  under  test. 

Although  eCPAS  version  1.3.0.14  wasn’t  fully 
functional  during  JEFX  09-3,  it  did  demonstrate  its 
potential  value  with  certain  features,  such  as  screen 
capture  technology  that  helped  the  assessor  to  record 
and  replay  an  operator’s  screens  during  the  experiment. 
Other  useful  features,  such  as  capturing  Link-16 
messages  and  displaying  JADOCS  remarks  and  mission 
history  information  in  a  timetable  with  the  unified 
search  function,  proved  their  worth  during  detailed  data 
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Table  1.  CPAS/eCPAS  capability  summary. 

Data  sources 

Information  provided 

CPAS  1.2*  JADOCS 

IWS 

Jabber  (XMPP) 
mIRC 

CPAS  1.3f  JADOCS 

IWS 

Jabber  (XMPP) 
mIRC 

Adobe  Connect 
Link- 16 
Voice 
Voice  text 
Video  capture 


Dynamic  targeting  mission  histories  (e.g.,  who,  what,  when,  where,  how) 
Dynamic  chat  content  (e.g.,  who,  when,  what) 


Dynamic  targeting  mission  histories  (e.g.,  who,  what,  when,  where,  how) 
Dynamic  chat  content  (e.g.,  who,  when,  what) 


Dynamic  data  link  content  (e.g.,  C2  system  messages  between  AOC  and  tactical  units) 
Recorded  .wav  files  of  tactical  communications 
Transcribed  files  of  tactical  voice  communications 

Single  view  and/or  sequenced  screen  shots  of  selected  user  work  station  displays 


IWS  =  Ezenia  Info  Workspace,  XMPP  =  extensible  messaging  and  presence  protocol,  mIRC  =  Madam-Bey  Internet  relay  chat. 

*CPAS  1.2  supports  individual  and  team  training.  The  system  correlates  AOC  dynamic  targeting  activities  with  chat,  exposing  activities  in 
context  to  exploit  learning  opportunities. 

JCPAS  1.3  supports  individual  and  team  training  and  operational  testing.  The  system  correlates  AOC  dynamic  targeting  activities  with 
multiple  interactive  chat  systems,  data  link  messages,  and  voice  transmissions  exposing  activities  in  a  rich  context  to  exploit  learning  opportunities 
and  generate  empirical  assessments  of  systems  under  test. 


analysis  (using  eCPAS  version  1.3.0.27)  after  JEFX  09- 
3.  The  scalable  timetable  feature  also  helped  assessors 
zoom  into  critical  timeframes,  such  as  when  messages 
are  sent  from  the  AOC  to  identify  potential  problems 
when  mission  scenarios  do  not  proceed  as  planned. 

In  summary,  the  CPAS  process  instrumentation 
systems  proved  useful  as  both  experiment  monitoring 
tools  and  analysis  augmentation  tools  for  WWF  II 
assessors.  These  CPAS  instantiations  still  require 
additional  functionality  to  serve  as  the  primary  analysis 
tools.  Specifically,  feedback  from  the  WWF  II 
assessors  indicates  the  need  to  expand  the  Link-16 
data  collection  to  other  J-messages,  as  well  as  to  pursue 
the  ability  to  collect  and  transcribe  voice  communica¬ 
tions  with  better  accuracy. 

Conclusion 

The  authors  believe  process  instrumentation,  which 
focuses  on  collecting  information  about  human  activities, 
provides  great  benefits  to  the  training  and  test  commu¬ 
nities.  Future  efforts  to  improve  and  enhance  the  current 
CPAS  capabilities  will  more  fully  realize  the  goal  of 
process  instrumentation  and  analysis.  JHU/APL  cur¬ 
rently  has,  or  is,  applying  the  concept  to  Air  Force  Air 
Support  Operations  Center  operations,  U.S.  Navy 
Maritime  Operations  Center  operations,  and  the  Joint 
Forces  Command  Joint  After- Action  Review  Resource 
Library.  In  addition,  JHU/APL  is  working  on  the  next 
iteration  of  the  C2  process  instrumentation  concept,  the 
Operational  Command  and  Control  Instrumentation 
System  (OC2IS,  pronounced  Oh-sis).  OC2IS  is  an 
FY10  Resource  Enhancement  Program  project  spon¬ 
sored  by  the  Office  of  the  Secretary  of  Defense’s  Test 


Resource  Management  Center  and  managed  by  the  Air 
Force  505th  Command  and  Control  Wing.  Although 
requirements  for  OC2IS  are  not  yet  finalized,  potential 
enhancements  include  distributed  data  collection,  remote 
data  collector  control,  Web-based  user  interfaces,  a 
service-based  architecture,  more  robust  voice  transcrip¬ 
tion,  automated  analyses  capabilities,  workflow  tracking, 
improved  user  configurations,  more  powerful  reconstruc¬ 
tion  tools,  and  collectors  for  new  data  sources.  By 
capturing  data  on  the  human  view  and  perspective  during 
process  execution  and  then  correlating  that  data  to  other 
data  to  determine  the  overall  impact  of  the  human-in- 
the-loop,  OC2IS  will  contribute  to  the  assessment  of 
how  well  systems  under  test  are  meeting  mission 
effectiveness  requirements. 

Along  with  developing  technology  to  collect  new 
types  of  data,  JHU/APL  is  also  researching  how  to  best 
portray  these  data  to  provide  meaningful  insight  into 
the  processes  and  the  human  component  of  total 
system  performance.  By  providing  a  better  understand¬ 
ing  of  human  activities  and  their  effects,  JHU/APL 
wants  to  develop  capabilities  to  evaluate  the  impacts  to 
C2  and  TTP  of  new  technologies  and  training 
methodologies.  This  includes  tracing  threads  of  human 
and  technology  activities,  as  well  as  evaluating  human 
aspects,  such  as  workload,  situation  awareness,  and 
decision  making  for  individuals,  and  the  collaboration 
and  coordination  of  teams.  □ 
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